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Abstract 

In recent years greater attention has been paid to develop learning outcomes for academic programs 
and then to develop methods to assess these learning outcomes. Generally speaking, there are two 
kinds of outcomes: course outcomes and program outcomes. Assessments of these learning outcomes 
in institutions of higher education are mandated by the accrediting organizations. This paper describes 
a methodology used by a Computer Information Systems program in a small undergraduate institution 
to develop its learning outcomes, to collect assessment data, and to evaluate or assess its course and 
program outcomes during a ten year period. The data collection and the subsequent data analysis 
showed the strengths and weaknesses of the program and we were able to address a number of these 
weaknesses. 

Key words: Course outcomes, Learning outcomes, Learning outcome assessments, Measurement, 
Outcome based education, Program outcomes, and Programs metrics. 


1. INTRODUCTION 

In recent years greater attention has been paid 
to develop learning outcomes for academic 
programs and then to develop meaningful 
assessment methods to evaluate these 
outcomes. Assessment is a systematic and on¬ 
going process of collecting, interpreting, and 
acting on information relating to the goals and 
outcomes developed to support the mission and 
purpose of an institution (Osters, 2003). 
According to Acharya (2003), assessments 
should help us to answer the following 
questions: (1) What do we want the students to 
learn? (2) Why do we want them to learn it? (3) 
How can we help them to learn it? (4) How do 
we know what they have learned? Also Osters 
(2003) pointed out that assessments should help 
us to improve what we are doing. Assessment 
begins with the articulation and development of 
measurable outcomes. Generally speaking, there 
are two kinds of learning outcomes: course 
outcomes and program outcomes. The course 
outcomes should describe what students are 


expected to learn from an individual course, 
while program outcomes should describe what a 
student is expected to accomplish after 
completing the coursework from the program. 
Maki (2002) pointed out that learning outcome 
assessments must be based on institutional 
curiosity to seek answers to questions about 
student learning, why they learn, how well they 
learn, when they learn, and explores how 
pedagogies and educational experiences 
develop, and foster student learning. Maki 
(2002) also pointed out that innovations in 
pedagogy or integration of diverse methods of 
teaching and learning into a program of study, 
redesign of a program, reconceptualizing the 
role of advising, or establishing stronger 
connections between curriculum and non¬ 
curriculum represents some of the kinds of 
changes that faculty and staff may undertake to 
improve student learning and development 
based on their interpretations of learning 
outcome assessment results. 
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2. PROGRAM LEARNING OUTCOME 
DEVELOPMENT 

Learning outcomes should describe what 
students will be able to demonstrate in terms of 
knowledge, skills, and values upon completion of 
a course, a span of several courses, or a degree 
program (Osters, 2003). Clear statement of 
learning outcomes serves as the foundation to 
assess the effectiveness of the teaching and 
learning process. According to Osters (2003), 
the three essential components of a measurable 
learning outcome are: (1) Student learning 
behaviors, (2) Appropriate assessment methods, 
and (3) Specific student performance criteria. 
Student behaviors describe what students are 
expected to demonstrate by the completion of 
the course. Action verbs like demonstrate, 
apply, define, analyze, etc. are used to describe 
student behaviors. Assessment methods are 
tools and techniques used to determine the 
extent to which the stated learning outcomes 
are achieved. Student performance criteria 
should be expressed in specific and measurable 
terms that are acceptable to a specific course or 
series of courses. A variety of methods, 
qualitative and quantitative, direct and indirect, 
should be used to assess the learning outcomes. 
Keep in mind that a simple letter grade alone 
does not provide adequate feedback to student's 
performance, because the letter grade alone 
does not sufficiently identify with the strengths 
and weaknesses of individual learning outcomes. 
If the grading system is accompanied by a rubric 
where the individual outcome components are 
addressed, then this tool can be used to pinpoint 
the weakness and strengths of the student's 
performance. 

It is very important to define the learning 
outcomes of a program/course in specific and 
precise manner. Spady & Marshall (1994) wrote: 

"Outcomes are clear, observable 
demonstrations of student learning that 
occur after a significant set of learning 
experiences...Typically these demonstrations, 
or performances, reflect three things: (1) 
what the student knows; (2) what the 
student can actually do with what he or she 
knows; (3) the student's confidence and 
motivation in carrying out the 
demonstration. A well-defined outcome will 
have clearly defined content or concepts and 
be demonstrated through a well-defined 
process beginning with directive or requests 
such as explain, organize, or produce." 
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After an exhaustive research the faculty 
members developed a number of outcomes for 
the CIS program and from this list we were able 
to select six measurable outcomes for our 
program. The American Association of Higher 
Education's (AAHH) (1996) nine principles of 
good practices for assessing student learning 
were used in the selection process. We also used 
a number of other research documents from the 
AAHE's assessment web site. Our hope is that 
the graduates of our program will be able to 
show that they have accomplished these six 
outcomes by receiving a degree from the CIS 
program. The following list shows the learning 
outcomes developed by the CIS program. 


1. Students will demonstrate the skill to 
write complete, complex programs that 
are fully tested. 

2. Students will demonstrate the skill to 

develop a complete information system 
that incorporates feasibility study, 
analysis, design, systems development, 
testing, implementation and 

maintenance. 

3. Students will demonstrate the ability to 
solve problems using the computer as a 
tool, using either application packages or 
custom programs. 

4. Students will demonstrate the ability to 
work as a team member in a problem¬ 
solving situation. 

5. Students will demonstrate the ability to 
investigate existing literature in 
Information systems. 

6. Students will demonstrate the ability to 
communicate effectively. 


Fig 1 

3. COURSE LEARNING OUTCOME 
DEVELOPMENT 

Once these outcomes were developed, we set 
out to see how these outcomes can be 
accomplished through our course offerings. We 
know that we have to develop a set of outcomes 
for each of our courses, keeping in mind that 
there must be a match between these course 
outcomes and the program outcomes. In other 
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words, the stated program outcomes must be 
accomplished through the course outcomes. 
Faculty who are teaching the individual courses 
are asked to take the program outcomes and 
see how these outcomes can be accomplished 
through their courses. Also these are the 
outcomes a faculty would like his/her students 
to know at the completion of that particular 
course. Axelsson and Melin (2010) pointed out 
that when learning outcomes are developed in a 
transparent and clear way, students will be able 
to use them before, during and after the course. 
The importance of measurability and clarity of 
the course outcomes were emphasized. Faculty 
members developed a set of learning outcomes 
for each course from which we selected five or 
six outcomes for each individual course. We then 
developed a table to show the relationship 
between program outcomes and courses 
offerings. We also agreed that when we develop 
new courses in the future, we need to pay 
greater attention to the course outcomes to see 
how the new course will satisfy the program 
outcomes. By adding new rows in Table 2 we 
will be able to get a quick view of the 
relationship between the course and program 
outcomes. 


Course 

Number 

Course Title 

CIS 119 

Visual Basic Programming 

CIS 218 

Introduction to Information 

Systems 

CIS 252 

Introduction to C++ Proqramminq 

CIS 260 

Cobol Proqramminq 

CIS 353 

Systems Analysis 

CIS 363 

Data Base Structures 

CIS 443 

Data Communication 

CIS 465 

Management Information Systems 

CIS 495 

Senior Project 

Electives 


CIS 352 

Data Structures Usinq C+ + 

CIS 340 

Java Proqramminq 

CIS 370 

Network Operatinq Systems 

CIS 455 

Computer Hardware & Software 

CIS 460 

Web Development 

CIS 470 

Information Assurance 

CIS 480 

Internships 

CIS 485 

Emerqinq Technoloqy 


Table 1 


Required 

Courses 

Learning Outcomes 

1 

2 

3 

4 

5 

6 

CIS 119 

X 


X 



X 

CIS 218 



X 

X 

X 

X 

CIS 252 

X 


X 



X 

CIS 260 

X 


X 



X 

CIS 353 


X 

X 

X 

X 

X 

CIS 363 


X 

X 

X 

X 

X 

CIS 443 



X 

X 

X 

X 

CIS 465 



X 

X 

X 

X 

CIS 495 

X 

X 

X 


X 

X 

Electives 


CIS 352 

X 


X 



X 

CIS 340 

X 


X 



X 

CIS 370 



X 

X 

X 

X 

CIS 455 



X 

X 

X 

X 

CIS 460 


X 

X 

X 

X 

X 

CIS 470 



X 

X 

X 

X 

CIS 480 



X 


X 

X 

CIS 485 



X 


X 

X 


Table 2 


The current CIS course offerings (course 
numbers and corresponding course titles) are 
listed in Table 1 for reference. Table 2 shows a 
mapping of the courses and the CIS program 
outcomes. 


Upon completion of this course, students will 
be able to demonstrate proficiency in: 

1. A disciplined approach to problem solving 
methods and algorithm development (CIS- 
0#1, 3) 

2. The syntax and vocabulary of Visual Basic.Net 
(CIS-0 # 1) 

3. The usage of Visual Basic.Net Programming 
Environment (CIS-0 #1) 

4. Developing complete Visual Basic programs 
that include specification, design, code, 
debugging, testing, and documentation. (CIS- 
O #1) 

5. Using computers as a tool in problem solving 
(CIS-0 # 3) 

6. Communicating the program development 
process in a predetermined format (CIS-0 #6) 


Fig 2 
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As mentioned earlier, we have developed a 
number of outcomes for each course. Faculty 
members also developed a number of rubrics for 
each course to assess the achievement of each 
student. 

The outcomes developed for CIS 119 (Visual 
Basic Programming) are given above. Similar 
outcomes were developed for all the other 
courses in our curriculum. 

4. EARLY LEARNING OUTCOMES 
ASSESSMENT 

Data collection and analysis of the course 
outcomes for each course is described in a 
previous paper (Abraham, 2006). This paper will 
be concentrating on the data collection and 
analysis for a period of ten years. At the end of 
each academic year we consolidate the data 
collected during the Fall and Winter semesters 
and report back to the academic dean who 
collects the data for accreditation and program 
assessment purposes. Please see appendix B for 
a sample form for this reporting. This form is 
developed by the office of the academic dean 
and used by all programs/departments in the 
University for uniform reporting. Column one 
states the outcomes while column two lists the 
course that satisfy the outcome and these 
course come from the previous mapping (table 
2). Column three lists the activities that a 
student will perform to satisfy the fulfillment of 
the outcome. Column four lists the percentages 
of students who met or exceeded the outcome 
expectations. The last column lists the actions 
taken by the program to address the issues 
raised during the data analysis. Columns one, 
two and 3 are self-explanatory while columns 
four and five need some explanation. We have 
made two assumptions to generate the data in 
column four. It is assumed that if a student has 
received more than 70% for an outcome then 
he/she has met the requirements for that 
outcome. For example if a student has received 
an aggregate of more than 70 % for the 
program development (outcome 1) then that 
student has met the requirements for outcome 
1. It is assumed that if more than 80% of the 
students in a class met the requirements for an 
outcome, then that class has met the 
requirements for that outcome. For example if 
more than 80% of the students in Visual Basic 
Class (CIS 119) have met the outcome 1 
requirements, then the whole class has met the 
requirements for outcome 1. The 70% and 80% 
guidelines are quite arbitrary and we thought 
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that those numbers are suitable for our purpose. 
Only the students who have completed the 
course are included in this analysis. As 
mentioned elsewhere, we are a small institution 
and thus all courses are not offered all 
semesters. So the outcome assessment data 
collected every year may consist of data from 
uneven offerings. For example we offer CIS 119 
(Visual Basic Programming) every semester 
while CIS 340 (Java Programming) will be 
offered only once in a year. To make things 
more difficult we offer some courses only once in 
two years. 

The outcome analysis revealed a lot of strengths 
and weakness of our course offerings and course 
delivery. Colum five lists the changes we have 
already implemented or the changes we are 
planning to implement as a result of the yearly 
outcome analysis. Some of these changes are 
easy to implement while some others need 
budget support from administration. For 
example we were able to emphasis the 
importance of more team work in all upper level 
courses as a result of the evaluation in early 
years. Some of the changes require hardware 
and software implementation and these kinds of 
changes need support from the administration. 

5. TEN YEAR ASSESSMENT 

We have been collecting and reporting the yearly 
outcome assessment data to the academic dean 
for almost ten years. Last year we had an 
accreditation visit from the North Central and 
thus we were asked to produce a five/ten year 
report of our program outcome assessment to 
be included in the final self-study report. The 
faculty members from the CIS program 
generated a ten year outcome assessment 
report. I am very happy to report that the 
visiting team was very impressed with the 
progress we have made in the outcome 
collection and analysis and we have received our 
10 year unconditional accreditation. 

We are a small institution with a low but steady 
enrollment and thus 100 and 200 level classes 
are offered every semester while 300 and 400 
level classes are offered in a two year cycle. 
Data from all classes are collected every 
semester and combined into an annual report to 
the dean. For the ten year report we combined 
all these annual data into one document. A copy 
of this report is presented in appendix A. 
Column one of this report restates the six 
program outcomes and column two lists the 
course that will satisfy these outcomes as 
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described for table 4 above. Column three lists 
the average for each course for the ten year 
period. This average is generated from the 
yearly data that is reported to the administration 
at the end of each academic year. The ten year 
average is calculated for each course and then 
for each outcome and listed in column 4 of table 

6. As stated before the 100 and 200 level 
classes are offered more times than the 300 and 
400 level classes and thus they have a greater 
effect on the ten year outcome average. We also 
calculated a final average by taking the average 
for all the six outcomes. In its current form the 
report shows that we as a program are doing 
well and our students are satisfying the stated 
program outcomes well. 

6. LESSONS LEARNED 

To draw reasonable conclusions from learning 
outcome assessments, we should make our 
assessments as fair as possible. Lam (1995) 
pointed out that a fair assessment is one in 
which students are given equitable opportunities 
to demonstrate what they know. Suskie (2000) 
suggested the following steps to make our 
assessments methods as fair as possible: (1) 
Have clearly stated learning outcomes and share 
them with your students, so they know what you 
expect from them, (2) Match your assessment to 
what you teach and vice versa, (3) Use different 
measures and many different kinds of measures, 
(4) Help students learn how to do the 
assessment tasks, (5) Engage and encourage 
your students, (6) Interpret assessment results 
appropriately, (7) Evaluate the outcomes of your 
assessments. 

Learning outcome assessment must be an 
ongoing process. According to Rodrigues (2002), 
assessment must become a part of an 
institution's culture. We have been doing these 
assessments for almost ten years now. The 
faculty in our program felt that the experience of 
going through the process was very worthwhile, 
even though it was very time consuming and 
frustrating. As a result of the data collection 
and analysis we were able to correct a number 
of problems in our course offerings as they 
occurred. The data collection and the 
subsequent data analysis show our strengths 
and weaknesses and we were able to address 
these issues in a timely manner. We strongly 
believe that the process of data collection and 
analysis is more important than the final number 
to understand what is happening in our program 
at any given point. 
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We used a number of other assessment 
techniques other than those described in this 
paper. All our graduating senior students are 
required to attend an exit interview. During the 
interview, a faculty member and the student 
address the program and course outcomes and 
solicit recommendation from the students. In 
addition to oral, written, and poster 
presentations, faculty members usually visit 
internship sites to evaluate the performance of 
the student interns. 

7. CONCLUSIONS 

Outcome based education promises a better way 
of understanding student learning, and in turn 
provide ways to improve the quality of 
education. To measure or assess the learning 
outcomes effectively, we need to start with 
measurable, concise, and specific learning 
outcomes for our program and individual course 
that must be shared and explained to the 
students. Clear and concise measuring tools, 
techniques, instruments, and methods must also 
be developed and must be conveyed to the 
students to avoid confusion and frustration. 
Assessment data must be collected in an 
ongoing basis using multiple methods and 
instruments. Collected data must be analyzed to 
understand the strengths and weaknesses of the 
program, courses, teaching, and learning. This 
information must be used to improve teaching 
and learning, incorporate innovations in 
pedagogy, redesign programs and courses, 
redevelopment of the outcomes, and the 
development of new tools for assessment. For 
outcome assessment to be successful it must be 
ongoing and must be part of the institution's 
culture. Administrators must recognize the 
importance of this process by providing financial 
and collateral support. Outcome based education 
is here to stay and it is important for educators 
to be prepared to accept the challenge of 
developing measurable outcomes for their 
programs/institutions, assess these outcomes, 
and then use the assessment data to improve 
what they are doing. 
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Appendix A 

Ten Year Outcome Summary 


Outcomes 

Courses 

10 Year Course 

10 year 

Averages 

Average 

Students will demonstrate the skill to write 

CIS 119 

83.25% 


complete, complex programs that are fully tested. 

CIS 252 

82.50% 



CIS 340 

85.00% 

88.35% 


CIS 460 

95.00% 



CIS 495 

96.00% 


Students will demonstrate the skill to develop a 

CIS 353 

97.50% 


complete information system including feasibility 

CIS 363 

94.00% 

95.13% 

study, analysis, design, systems development, 

CIS 460 

94.00% 

testing, implementation and maintenance. 

CIS 495 

95.00% 


Students will demonstrate the ability to solve 

CIS 119 

83.75% 


problems using the computer as a tool, using 

CIS 218 

80.50% 


either application packages or custom programs. 

CIS252 

84.00% 



CIS 340 

84.00% 



CIS 363 

90.00% 

90.75% 


CIS 370 

90.00% 



CIS 460 

95.00% 


Students will demonstrate the ability to work as a 

CIS 218 

85.75% 


team in a problem-solving situation. 

CIS 353 

95.00% 



CIS 363 

95.00% 



CIS 370 

88.00% 

91.17% 


CIS 460 

89.00% 



CIS 465 

94.25% 


Students will demonstrate the ability to 

CIS 218 

82.00% 


investigate existing literature in Information 

CIS 353 

86.00% 


systems 

CIS 363 

88.00% 



CIS 370 

85.00% 

88.71% 


CIS 460 

90.00% 


CIS 465 

94.00% 



CIS 495 

96.00% 


Students will demonstrate the ability to 

CIS 119 

82.75% 


communicate effectively. 

CIS 218 

82.00% 



CIS 252 

84.50% 



CIS 340 

84.00% 



CIS 353 

90.00% 



CIS 363 

90.00% 

89.52% 


CIS 443 

95.00% 



CIS 460 

90.50% 



CIS 495 

96.00% 


10 Year Program Average 

90.60% 
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